\documentclass{article} % For LaTeX2e
\usepackage{iclr2019_conference,times}

% Optional math commands from https://github.com/goodfeli/dlbook_notation.
\input{math_commands.tex}

\usepackage{hyperref}
\usepackage{url}
\usepackage{booktabs}
\usepackage{tabularx}
\usepackage{multirow}
\usepackage{graphicx}


\title{GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding}

% Authors must not appear in the submitted version. They should be hidden
% as long as the \iclrfinalcopy macro remains commented out below.
% Non-anonymous submissions will be rejected without review.

\author{Alex Wang$^1$, Amanpreet Singh$^1$, Julian Michael$^2$, Felix Hill$^3$, \\ \textbf{Omer Levy$^2$ \& Samuel R. Bowman$^1$} \\
$^1$Courant Institute of Mathematical Sciences, New York University\\
$^2$Paul G. Allen School of Computer Science \& Engineering, University of Washington\\
$^3$DeepMind\\
\texttt{\{alexwang,amanpreet,bowman\}@nyu.edu} \\
\texttt{\{julianjm,omerlevy\}@cs.washington.edu} \\
\texttt{felixhill@google.com}
}

% \author{Alex Wang, Amanpreet Singh \& Samuel R. Bowman \\
% Courant Institute of Mathematical Sciences\\
% New York University\\
% \texttt{\{alexwang,amanpreet,bowman\}@nyu.edu} \\
% \AND
% Julian Michael \& Omer Levy \\
% Paul G. Allen School of Computer Science \& Engineering \\
% University of Washington \\
% \texttt{\{julianjm,omerlevy\}@cs.washington.edu} \\
% \And
% Felix Hill \\
% DeepMind \\
% London, UK \\
% \texttt{felixhill@google.com}
% }

% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.

\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\iclrfinalcopy % Uncomment for camera-ready version, but NOT for submission.
\begin{document}


\maketitle

\begin{abstract}
For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset.
In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks.
By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks.
GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models.
We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems. 
\end{abstract}


\section{Introduction}\label{sec:intro}

The human ability to understand language is \emph{general}, \textit{flexible}, and \textit{robust}. 
% We can effectively interpret and respond to utterances of diverse form and function in many different contexts. % SB: We don't really want to talk about context here, as that'll cue readers to ask about sentence understanding in broader contexts than we test.
In contrast, most NLU models above the word level are designed for a specific task and struggle with out-of-domain data. 
If we aspire to develop models with understanding beyond the detection of superficial correspondences between inputs and outputs, 
then it is critical to develop a more unified model that can learn to execute a range of different linguistic tasks in different domains.

%To motivate research in this direction, we present the General Language Understanding Evaluation benchmark (GLUE, \href{https://gluebenchmark.com}{\tt gluebenchmark.com}), an online tool for evaluating the performance of a single NLU model across multiple tasks, including question answering, sentiment analysis, and textual entailment, built largely on established existing datasets.
To facilitate research in this direction, we present the General Language Understanding Evaluation 
%(GLUE, \href{https://gluebenchmark.com}{\tt gluebenchmark.com}) 
(GLUE)
benchmark: a collection of NLU tasks including question answering, sentiment analysis, and textual entailment, and an associated online platform for model evaluation, comparison, and analysis.
GLUE does not place any constraints on model architecture beyond the ability to process single-sentence and sentence-pair inputs and to make corresponding predictions. 
For some GLUE tasks, training data is plentiful, but for others it is limited or fails to match the genre of the test set. GLUE therefore favors models that can learn to represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks. 
None of the datasets in GLUE were created from scratch for the benchmark; we rely on preexisting datasets because they have been implicitly agreed upon by the NLP community as challenging and interesting.
Four of the datasets feature privately-held test data, which will be used to ensure that the benchmark is used fairly.\footnote{To evaluate on the private test data, users of the benchmark must submit to \texttt{\href{gluebenchmark.com}{gluebenchmark.com}}}
% SB: We mention private data in the contributions list, so it should appear first here. Here's my attempt.

%Though GLUE is designed to prefer models with general and robust language understanding, we cannot entirely rule out the existence of simple superficial strategies for solving any of the included tasks. 
To understand the types of knowledge learned by models and to encourage linguistic-meaningful solution strategies, GLUE also includes a set of hand-crafted analysis examples for probing trained models. 
%Unlike many test sets employed in machine learning research that reflect the frequency distribution of naturally occurring data or annotations, 
%This dataset is designed to highlight points of difficulty that are relevant to model development and training, such as the incorporation of world knowledge or the handling of lexical entailments and negation. 
This dataset is designed to highlight common challenges, such as the use of world knowledge and logical operators, that we expect models must handle to robustly solve the tasks.

To better understand the challenged posed by GLUE, we conduct experiments with simple baselines and state-of-the-art sentence representation models.
We find that unified multi-task trained models slightly outperform comparable models trained on each task separately.
Our best multi-task model makes use of ELMo \citep{peters2018deep},
a recently proposed pre-training technique.
However, this model still achieves a fairly low absolute score.
%, indicating room for improved general NLU systems.
Analysis with our diagnostic dataset reveals that our baseline models deal well with strong lexical signals but struggle with deeper logical structure.

In summary, we offer: (i) A suite of nine sentence or sentence-pair NLU tasks, built on established annotated datasets and selected to cover a diverse range of text genres, dataset sizes, and degrees of difficulty. (ii) An online evaluation platform and leaderboard, based primarily on privately-held test data. The platform is model-agnostic, and can evaluate any method capable of producing results on all nine tasks. (iii) An expert-constructed diagnostic evaluation dataset. (iv) Baseline results for several major existing approaches to sentence representation learning.



\begin{table*}[t]
\centering \small
\begin{tabular}{lrrlll}
 \toprule
\textbf{Corpus} & \textbf{$|$Train$|$} & \textbf{$|$Test$|$} & \textbf{Task} & \textbf{Metrics} & \textbf{Domain} \\
\midrule
\multicolumn{6}{c}{Single-Sentence Tasks}\\
\midrule
CoLA & 8.5k & \textbf{1k} & acceptability & Matthews corr.& misc. \\ % SB: Changed from 'linguistics literature'. That could be misleading, as few of the sentences are actually in the style of academic writing, and many are found in the wild.
SST-2 & 67k & 1.8k & sentiment & acc. & movie reviews \\
\midrule
\multicolumn{6}{c}{Similarity and Paraphrase Tasks}\\
\midrule
MRPC & 3.7k & 1.7k & paraphrase & acc./F1 & news \\
STS-B & 7k & 1.4k & sentence similarity & Pearson/Spearman corr. & misc. \\
QQP & 364k & \textbf{391k} & paraphrase & acc./F1 & social QA questions \\
\midrule
\multicolumn{6}{c}{Inference Tasks} \\
\midrule
MNLI & 393k & \textbf{20k} & NLI & matched acc./mismatched acc. & misc. \\
QNLI & 105k & 5.4k & QA/NLI & acc. & Wikipedia \\
RTE & 2.5k & 3k & NLI & acc. & news, Wikipedia \\
WNLI & 634 & \textbf{146} & coreference/NLI & acc. & fiction books \\
\bottomrule
\end{tabular}
\caption{Task descriptions and statistics. All tasks are single sentence or sentence pair classification, except STS-B, which is a regression task. MNLI has three classes; all other classification tasks have two. Test sets shown in bold use labels that have never been made public in any form.
}
\label{tab:tasks}
\end{table*}

\section{Related Work}
\label{sec:related}
\citet{collobert2011natural} used a multi-task model with a shared sentence understanding component to jointly learn POS tagging, chunking, named entity recognition, and semantic role labeling.
% TODO: Worth adding a citation or two here to show that work on pure multitask is still active, not just the odd hybrid stuff like Hashimoto. Something from Bingel? If we do that, we can probably drop the below.
More recent work has explored using labels from core NLP tasks to supervise training of lower levels of deep neural networks \citep{sogaard2016deep,hashimoto2016joint} and 
automatically learning cross-task sharing mechanisms for multi-task learning \citep{ruder2017sluice}.
%identifying the conditions under which multi-task learning is useful \citep{bingel2017identifying}.

Beyond multi-task learning, much work in developing general NLU systems has focused on sentence-to-vector encoders \citep[][ i.a.]{pmlr-v32-le14,kiros2015skip}, leveraging unlabeled data 
%dai2015semi,kiros2015skip,jernite2017discourse,DBLP:journals/corr/abs-1710-04334,P17-1161,logeswaran2018efficient,
\citep{hill2016learning,peters2018deep}, labeled data \citep{conneau2018senteval,mccann2017learned}, and combinations of these \citep{collobert2011natural,subramanian2018large}.
In this line of work, a standard evaluation practice has emerged, recently codified as SentEval \citep{DBLP:conf/emnlp/ConneauKSBB17,conneau2018senteval}.
Like GLUE, SentEval relies on a set of existing classification tasks involving either one or two sentences as inputs. Unlike GLUE, SentEval only evaluates sentence-to-vector encoders, making it well-suited for evaluating models on tasks involving sentences \emph{in isolation}.
%Specifically, SentEval feeds the output of a pre-trained sentence encoder into lightweight task-specific models (typically linear classifiers) that are trained and tested on task-specific data.
However, cross-sentence contextualization and alignment are instrumental in achieving state-of-the-art performance on tasks such as machine translation \citep{bahdanau2014neural,vaswani2017attention}, question answering \citep{seo2016bidirectional}, and natural language inference \citep{rocktaschel2015reasoning}.
%\footnote{In the case of SNLI \citep{bowman2015large}, the best-performing sentence encoding model on the leaderboard as of April 2018 achieves 86.3\% accuracy, while the best performing attention-based model achieves 89.3\%.}.
GLUE is designed to facilitate the development of these methods: It is model-agnostic, allowing for any kind of representation or contextualization, including models that use no explicit vector or symbolic representations for sentences whatsoever.
%Indeed, among our baseline models the use of attention consistently leads to improved performance on GLUE.

GLUE also diverges from SentEval in the selection of evaluation tasks that are included in the suite. Many of the SentEval tasks are closely related to sentiment analysis, such as MR \citep{pang2005seeing}, SST \citep{socher2013recursive}, CR \citep{hu2004mining}, and SUBJ \citep{pang2004sentimental}. Other tasks are so close to being solved that evaluation on them is relatively  uninformative, such as MPQA \citep{wiebe2005annotating} and TREC question classification \citep{voorhees1999trec}. In GLUE, we attempt to construct a benchmark that is both diverse and difficult.


%In work which appeared after the initial launch of GLUE, 
\citet{McCann2018decaNLP} introduce decaNLP, which also scores NLP systems based on their performance on multiple datasets. Their benchmark recasts the ten evaluation tasks as question answering, converting tasks like summarization and text-to-SQL semantic parsing into question answering using automatic transformations. That benchmark lacks the leaderboard and error analysis toolkit of GLUE, but more importantly, we see it as pursuing a more ambitious but less immediately practical goal: While GLUE rewards methods that yield good performance on a circumscribed set of tasks using methods like those that are currently used for those tasks, their benchmark rewards systems that make progress toward their goal of unifying all of NLU under the rubric of question answering.

% Note: Moved table to 01intro.tex for better placement. -SB

\section{Tasks}\label{sec:tasks}

GLUE is centered on nine English sentence understanding tasks, which  cover a broad range of domains, data quantities, and difficulties.
As the goal of GLUE is to spur development of generalizable NLU systems, we design the benchmark such that good performance should require a model to share substantial knowledge (e.g., trained parameters) across all tasks, while still maintaining some task-specific components.
Though it is possible to train a single model for each task with no pretraining or other outside sources of knowledge and evaluate the resulting set of models on this benchmark, 
we expect that our inclusion of several data-scarce tasks will ultimately render this approach uncompetitive.
We describe the tasks below and in \autoref{tab:tasks}.  Appendix \ref{sec:apdx_data} includes additional details. Unless otherwise mentioned, tasks are evaluated on accuracy and are balanced across classes.

\subsection{Single-Sentence Tasks}

\paragraph{CoLA}
The Corpus of Linguistic Acceptability \citep{warstadt2018neural}
%consists of examples of expert English sentence acceptability judgments drawn from 22 books and journal articles on linguistic theory.
consists of English acceptability judgments drawn from books and journal articles on linguistic theory.
Each example is a sequence of words annotated with whether it is a grammatical English sentence. 
%Superficially, this data is similar to our analysis data in that it is constructed to demonstrate potentially subtle and difficult contrasts.
%Judgments of this particular kind are the primary form of evidence in syntactic theory \citep{schutze-96}, so a machine learning system capable of predicting them reliably would offer potentially substantial evidence on questions of language learnability and innate bias. 
Following the authors, we use Matthews correlation coefficient \citep{matthews1975comparison} as the evaluation metric, which evaluates performance on unbalanced binary classification and ranges from -1 to 1, with 0 being the performance of uninformed guessing.
We use the standard test set, for which we obtained private labels from the authors.
We report a single performance number on the combination of the in- and out-of-domain sections of the test set.

\paragraph{SST-2}
The Stanford Sentiment Treebank \citep{socher2013recursive} consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence. We use the two-way (\textit{positive}/\textit{negative}) class split, and use only sentence-level labels.

\subsection{Similarity and Paraphrase Tasks}

\paragraph{MRPC}
The Microsoft Research Paraphrase Corpus \citep{dolan2005automatically} is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent. Because the classes are imbalanced (68\% positive), we follow common practice and report both accuracy and F1 score.

\paragraph{QQP}
The Quora Question Pairs\footnote{ \href{https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs}{\texttt{data.quora.com/\allowbreak First-\allowbreak Quora-\allowbreak Dataset-\allowbreak Release-Question-Pairs}}} dataset is a collection of question pairs from the community question-answering website Quora. The task is to determine whether a pair of questions are semantically equivalent. As in MRPC, the class distribution in QQP is unbalanced (63\% negative), so we report both accuracy and F1 score. We use the standard test set, for which we obtained private labels from the authors. We observe that the test set has a different label distribution than the training set.

\paragraph{STS-B}
The Semantic Textual Similarity Benchmark \citep{cer2017semeval} is a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data. 
Each pair is human-annotated with a similarity score from 1 to 5; the task is to predict these scores.
Follow common practice, we evaluate using Pearson and Spearman correlation coefficients.

\subsection{Inference Tasks}

\paragraph{MNLI}
%The Multi-Genre Natural Language Inference Corpus \citep{DBLP:journals/corr/WilliamsNB17} is a crowd-sourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis, contradicts the hypothesis, or neither (\textit{neutral}). The premise sentences are gathered from a diverse set of sources, including transcribed speech, popular fiction, and government reports. The test set is broken into two sections: \textit{matched}, which is drawn from the same sources as the training set, and \textit{mismatched}, which uses different sources and thus requires domain transfer. We use the standard test set, for which we obtained labels privately from the authors, and evaluate on both sections. 
The Multi-Genre Natural Language Inference Corpus \citep{DBLP:journals/corr/WilliamsNB17} is a crowd-sourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (\textit{entailment}), contradicts the hypothesis (\textit{contradiction}), or neither (\textit{neutral}). The premise sentences are gathered from ten different sources, including transcribed speech, fiction, and government reports. We use the standard test set, for which we obtained private labels from the authors, and evaluate on both the \textit{matched} (in-domain) and \textit{mismatched} (cross-domain) sections. We also use and recommend the SNLI corpus \citep{bowman2015large} as 550k examples of auxiliary training data. %\citep{chen2017recurrent,gong2018nli}.

\paragraph{QNLI}
The Stanford Question Answering Dataset (\citealt{rajpurkar2016squad}) is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator). We convert the task into sentence pair classification by forming a pair between each question and each sentence in the corresponding context, and filtering out pairs with low lexical overlap between the question and the context sentence. The task is to determine whether the context sentence contains the answer to the question. This modified version of the original task removes the requirement that the model select the exact answer, but also removes the simplifying assumptions that the answer is always present in the input and that lexical overlap is a reliable cue.
This process of recasting existing datasets into NLI is similar to methods introduced in \citet{white2017inference} and expanded upon in \citet{demszky2018transforming}.
We call the converted dataset QNLI (Question-answering NLI).\footnote{An earlier release of QNLI had an artifact where the task could be modeled and solved as an easier task than we describe here. We have since released an updated version of QNLI that removes this possibility.}

\paragraph{RTE}
The Recognizing Textual Entailment (RTE) datasets come from a series of annual textual entailment challenges. We combine the data from RTE1 \citep{dagan2006pascal}, RTE2 \citep{bar2006second}, RTE3 \citep{giampiccolo2007third}, and RTE5 \citep{bentivogli2009fifth}.\footnote{RTE4 is not publicly available, while RTE6 and RTE7 do not fit the standard NLI task.} Examples are constructed based on news and Wikipedia text. 
We convert all datasets to a two-class split, where for three-class datasets we collapse \textit{neutral} and \textit{contradiction} into \textit{not\_entailment}, for consistency.
% The task is to predict if the premise entails the hypothesis. 

\begin{table*}[t]
\small
\centering
\begin{tabular}{ll}
\toprule
\textbf{Coarse-Grained Categories} & \textbf{Fine-Grained Categories} \\
\midrule
\multirow{2}{*}{Lexical Semantics} & Lexical Entailment, Morphological Negation, Factivity, \\ & Symmetry/Collectivity, Redundancy, Named Entities, Quantifiers \\
\midrule
\multirow{3}{*}{Predicate-Argument Structure} & Core Arguments, Prepositional Phrases, Ellipsis/Implicits, \\ & Anaphora/Coreference
Active/Passive, Nominalization, \\ & Genitives/Partitives, Datives, Relative Clauses, \\
& Coordination Scope, Intersectivity, Restrictivity \\
\midrule
\multirow{2}{*}{Logic} & Negation, Double Negation, Intervals/Numbers, Conjunction, Disjunction, \\ & Conditionals, Universal, Existential, Temporal, Upward Monotone, \\ & Downward Monotone, Non-Monotone \\
\midrule
Knowledge & Common Sense, World Knowledge\\

\bottomrule
\end{tabular}
\caption{The types of linguistic phenomena annotated in the diagnostic dataset, organized under four major categories. For a description of each phenomenon, see \autoref{sec:apdx_diagnostic}.}
\label{tab:analysis-categories}
\end{table*}

\paragraph{WNLI}
The Winograd Schema Challenge \citep{levesque2011winograd} is a reading comprehension task in which a system must read a sentence with a pronoun and select the referent of that pronoun from a list of choices. 
The examples are manually constructed to foil simple statistical methods: Each one is contingent on contextual information provided by a single word or phrase in the sentence. 
To convert the problem into sentence pair classification, we construct sentence pairs by replacing the ambiguous pronoun with each possible referent.
%The task (a slight relaxation of the original Winograd Schema Challenge) is to predict if the sentence with the pronoun substituted is entailed by the original sentence.
The task is to predict if the sentence with the pronoun substituted is entailed by the original sentence.
We use a small evaluation set consisting of new examples derived from fiction books\footnote{See similar examples at 
\href{https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html}{\tt cs.nyu.edu/\allowbreak faculty/\allowbreak davise/\allowbreak papers/\allowbreak WinogradSchemas/\allowbreak WS.html}} that was shared privately by the authors of the original corpus. 
While the included training set is balanced between two classes,  the test set is imbalanced between them (65\% not entailment). Also, due to a data quirk, the development set is \textit{adversarial}: hypotheses are sometimes shared between training and development examples, so if a model memorizes the training examples, they will predict the wrong label on corresponding development set example. As with QNLI, each example is evaluated separately, so there is not a systematic correspondence between a model's score on this task and its score on the unconverted original task.
We call converted dataset WNLI (Winograd NLI).

% analysis examples table
\begin{table*}[t]
\small
\centering
\begin{tabularx}{\textwidth}{XXXll}
\toprule
 \textbf{Tags} & \textbf{Sentence 1} & \textbf{Sentence 2} & \textbf{Fwd} & \textbf{Bwd} \\
\midrule
%\it Active/Passive (Predicate-Argument Structure) & Cape sparrows eat seeds, along with soft plant parts and insects. & Cape sparrows are eaten by seeds, along with soft plant parts and insects. & N & N \\ \midrule
%\it Symmetry/Collectivity (Lexical Semantics), Core Args (Predicate Argument Structure) & Tulsi Gabbard disagrees with Bernie Sanders on what is the best way to deal with Bashar al-Assad. & Tulsi Gabbard and Bernie Sanders disagree on what is the best way to deal with Bashar al-Assad. & E & E \\ \midrule
%\it Named Entities (Lexical Semantics), World Knowledge (Knowledge) & Musk decided to offer up his personal Tesla roadster. & Musk decided to offer up his personal car. & E & N \\ \midrule
\it Lexical Entailment (Lexical Semantics), Downward Monotone (Logic) & The timing of the meeting has not been set, according to a Starbucks spokesperson. & The timing of the meeting has not been considered, according to a Starbucks spokesperson. & N & E \\ \midrule
\it Universal Quantifiers (Logic) & Our deepest sympathies are with all those affected by this accident. & Our deepest sympathies are with a victim who was affected by this accident. & E & N \\ \midrule
\it Quantifiers (Lexical Semantics), Double Negation (Logic) & I have never seen a hummingbird not flying. & I have never seen a hummingbird. & N & E \\
\bottomrule
\end{tabularx}
\caption{Examples from the diagnostic set. \textit{Fwd} (resp. \textit{Bwd}) denotes the label when sentence 1 (resp. sentence 2) is the premise. Labels are \textit{entailment} (E), \textit{neutral} (N), or \textit{contradiction} (C).
Examples are tagged with the phenomena they demonstrate, and each phenomenon belongs to one of four broad categories (in parentheses).
%See \autoref{tab:analysis-categories} in Appendix \ref{sec:apdx_data} for a complete tag taxonomy.
}
\label{tab:analysis-examples}
\end{table*}



\subsection{Evaluation}
The GLUE benchmark follows the same evaluation model as SemEval and Kaggle. To evaluate a system on the benchmark, one must run the system on the provided test data for the tasks, then upload the results to the website \href{https://gluebenchmark.com}{\tt gluebenchmark.com} for scoring. 
%The site will then show an overall score for the main suite of tasks, and per-task scores on the main tasks and the diagnostic dataset. 
The benchmark site shows per-task scores and a macro-average of those scores to determine a system's position on the leaderboard.
For tasks with multiple metrics (e.g., accuracy and F1), we use an unweighted average of the metrics as the score for the task when computing the overall macro-average.
The website also provides fine- and coarse-grained results on the diagnostic dataset. See Appendix \ref{sec:apdx_website} for details.


%\subsection{Data and Bias}
%We do not endorse the use of the task training sets for any specific \textit{non-research} use. They do not cover every dialect of English one may wish to handle, nor languages other than English. As all of them contain text or annotations that were collected in uncontrolled settings, they contain evidence of stereotypes and biases that one may not wish one's system to learn \citep{rudinger2017social}. 


\section{Diagnostic Dataset}

Drawing inspiration from the FraCaS suite \citep{cooper96fracas} and the recent Build-It-Break-It competition \citep{ettinger2017towards}, we include a small, manually-curated test set for the analysis of system performance. While the main benchmark mostly reflects an application-driven distribution of examples, 
our diagnostic dataset highlights a pre-defined set of phenomena that we believe are interesting and important for models to capture. We show the full set of phenomena in \autoref{tab:analysis-categories}.

Each diagnostic example is an NLI sentence pair with tags for the phenomena demonstrated.
The NLI task is well-suited to this kind of analysis, as it can easily evaluate the full set of skills involved in (ungrounded) sentence understanding, from resolution of syntactic ambiguity to pragmatic reasoning with world knowledge.
We ensure the data is reasonably diverse by producing examples for a variety of linguistic phenomena and basing our examples on naturally-occurring sentences from several domains (news, Reddit, Wikipedia, academic papers).
This approaches differs from that of FraCaS, which was designed to test linguistic theories with a minimal and uniform set of examples.
A sample from our dataset is shown in \autoref{tab:analysis-examples}. 


% \paragraph{Linguistic Phenomena}
%We tag every example with fine- and coarse-grained categories of the linguistic phenomena they involve (categories shown in \autoref{tab:analysis-categories} in the appendix).
% The coarse-grained categories are \textit{Lexical Semantics}, \textit{Predicate-Argument Structure}, \textit{Logic}, and \textit{Knowledge and Common Sense}.
% Each has several fine-grained subcategories.
%While each example was collected with a single phenomenon in mind, a single example can be tagged with more than one category.
% For example, to know that \textit{I like some dogs} entails \textit{I like some animals}, it is not sufficient to know that \textit{dog} lexically entails \textit{animal}; one must also know that \textit{dog/animal} appears in an upward monotone context in the sentence. This example would be classified under both \textit{Lexical Semantics \(>\) Lexical Entailment} and \textit{Logic \(>\) Upward Monotone}.
% SB: Helpful but unnecessary. Can add back in CR.

%\paragraph{Domains} 
%We construct sentence pairs based on text from four domains:
%News (articles linked from the front page%\footnote{\url{news.google.com}}
%),
%Reddit (threads linked from the Front Page%\footnote{\url{reddit.com}}
%),
%Wikipedia (Featured Articles%\footnote{\url{en.wikipedia.org/wiki/Wikipedia:Featured_articles}}
%), 
%and academic papers from recent ACL conferences. We include 100 sentence pairs constructed from each source and 150 artificially-constructed sentence pairs for 550 total.
% SB: Can move to appendix if needed.

\paragraph{Annotation Process} 
We begin with a target set of phenomena, based roughly on those used in the FraCaS suite \citep{cooper96fracas}.
We construct each example by locating a sentence that can be easily made to demonstrate a target phenomenon, and editing it in two ways to produce an appropriate sentence pair.
%In many cases, we make these modifications small in order to encourage high lexical and structural overlap within each sentence pair and limit superficial cues.
We make minimal modifications so as to maintain high lexical and structural overlap within each sentence pair and limit superficial cues.
We then label the inference relationships  between the sentences, considering each sentence alternatively as the premise, producing two labeled examples for each pair (1100 total).
Where possible, we produce several pairs with different labels for a single source sentence, to have minimal sets of sentence pairs that are lexically and structurally very similar but correspond to different entailment relationships.
%After an initial round of annotation, we revise to include phenomena we think can be located in the data and collapse categories that are difficult to differentiate.
%We then set a minimum number of examples for each fine-grained category in each domain and use these counts to guide data collection (making sure to surpass the minimum) to ensure a baseline amount of diversity in the dataset.
% After finalizing the categories, we gathered a minimum number of examples in each fine-grained category from each domain to ensure a baseline level of diversity.
The resulting labels are 42\% \textit{entailment}, 35\% \textit{neutral}, and 23\% \textit{contradiction}.

\paragraph{Evaluation}
Since the class distribution in the diagnostic set is not balanced, we use \(R_3\) \citep{gorodkin2004Rk}, a three-class generalization of the Matthews correlation coefficient, for evaluation.
% This coefficient was introduced by~\newcite{gorodkin2004Rk} as \(R_K\), a generalization of the Pearson correlation that works for \(K\) dimensions by averaging the square error from the mean value in each dimension, i.e., calculating the full covariance between the input and output. In the discrete case, it generalizes Matthews correlation, where a value of 1 means perfect correlation and 0 means random chance.
%\omer{Julian: can we define $R_3$ here *succinctly*? It's also OK to refer to give a brief description and then refer to some external source. We just want the paper to be as self-contained as possible.}
%\julian{Here's a full-ish description and citation for \(R_K\). we can still shorten it of course.}

% \paragraph{Validation}
In light of recent work showing that crowdsourced data often contains artifacts which can be exploited to perform well without solving the intended task 
\citep[][ i.a.]{schwartz17cloze,poliak2018hypothesis,TSUCHIYA18.786},
%we perform an audit of our manually curated data as a sanity check.
we audit the data for such artifacts.
We reproduce the methodology of \citet{gurudipta18artifacts},
training two fastText classifiers \citep{joulin2016bag} to predict entailment labels on SNLI and MNLI using only the hypothesis as input. 
The models respectively get near-chance accuracies of 32.7\% and 36.4\% on our diagnostic data, showing that the data does not suffer from such artifacts. 

To establish human baseline performance on the diagnostic set, we have six NLP researchers annotate 50 sentence pairs (100 entailment examples) randomly sampled from the diagnostic set. Inter-annotator agreement is high, with a Fleiss's \(\kappa\) of 0.73.
The average \(R_3\) score among the annotators is 0.80, much higher than any of the baseline systems described in Section \ref{sec:baselines}. 

%We also evaluate state-of-the-art NLI models on the diagnostic dataset and find their overall performance to be rather weak, further suggesting that no easily-gameable artifacts present in existing training data are abundant in the diagnostic dataset (see Section~\ref{sec:experiments}).

\paragraph{Intended Use}
The diagnostic examples are hand-picked to address certain phenomena, and NLI is a task with no natural input distribution, so we do not expect performance on the diagnostic set to reflect overall performance or generalization in downstream applications. Performance on the analysis set should be compared between models but not between categories. The set is provided not as a benchmark, but as an analysis tool for error analysis, qualitative model comparison, and development of adversarial examples.

%Because these analysis examples are hand-picked to address certain phenomena, we expect that they will not be representative of the distribution of language as a whole, even in the targeted domains. However, NLI is a task with no natural input distribution. We deliberately select sentences that we hope will be able to provide insight into what models are doing, what phenomena they catch on to, and where are they limited. This means that the raw performance numbers on the analysis set should be taken with a grain of salt. The set is provided not as a benchmark, but as an analysis tool to paint in broad strokes the kinds of phenomena a model may or may not capture, and to provide a set of examples that can serve for error analysis, qualitative model comparison, and development of adversarial examples that expose a model's weaknesses.


\section{Baselines}\label{sec:baselines}

For baselines, we evaluate a multi-task learning model trained on the GLUE tasks, as well as several variants based on recent pre-training methods.
We briefly describe them here. See Appendix \ref{sec:apdx_baselines} for details.
We implement our models in the AllenNLP library \citep{Gardner2017AllenNLPAD}.
Original code for the baselines is available at \texttt{\href{https://github.com/nyu-mll/GLUE-baselines}{https://github.com/nyu-mll/GLUE-baselines}} 
and a newer version is available at \texttt{\href{https://github.com/jsalt18-sentence-repl/jiant}{https://github.com/jsalt18-sentence-repl/jiant}}.

\paragraph{Architecture}

Our simplest baseline architecture is based on sentence-to-vector encoders, and sets aside GLUE's ability to evaluate models with more complex structures.
Taking inspiration from \citet{DBLP:conf/emnlp/ConneauKSBB17}, the model uses a two-layer, 1500D (per direction) BiLSTM with max pooling and 300D GloVe word embeddings \citep[840B Common Crawl version;][]{pennington2014glove}.
For single-sentence tasks, we encode the sentence and pass the resulting vector to a classifier.
For sentence-pair tasks, we encode sentences independently to produce vectors $u, v$, and pass $[u; v; |u - v|; u * v]$ to a classifier.
The classifier is an MLP with a 512D hidden layer.

We also consider a variant of our model which for sentence pair tasks uses an attention mechanism inspired by \citet{seo2016bidirectional} between all pairs of words, followed by a second BiLSTM with max pooling.
By explicitly modeling the interaction between sentences, these models fall outside the sentence-to-vector paradigm.

\paragraph{Pre-Training} We augment our base model with two recent methods for pre-training: ELMo and CoVe. 
We use existing trained models for both.

ELMo uses a pair of two-layer neural language models trained on the Billion Word Benchmark \citep{chelba2013one}. 
Each word is represented by a contextual embedding, produced by taking a linear combination of the corresponding hidden states of each layer of the two models. 
We follow the authors' recommendations\footnote{\href{https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md}{\tt github.com/\allowbreak allenai/\allowbreak allennlp/\allowbreak blob/\allowbreak master/\allowbreak tutorials/\allowbreak how\textunderscore to/\allowbreak elmo.md}} and use ELMo embeddings in place of any other embeddings.

CoVe \citep{mccann2017learned} uses a two-layer BiLSTM encoder originally trained for English-to-German translation. 
The CoVe vector of a word is the corresponding hidden state of the top-layer LSTM. 
As in the original work, we concatenate the CoVe vectors to the GloVe word embeddings. 

\paragraph{Training}

We train our models with the BiLSTM sentence encoder and post-attention BiLSTMs shared across tasks, and classifiers trained separately for each task.
For each training update, we sample a task to train with a probability proportional to the number of training examples for each task.
We train our models with Adam \citep{kingma2014adam} with initial learning rate $10^{-4}$ and batch size 128.
We use the macro-average score as the validation metric and stop training when the learning rate drops below $10^{-5}$ or performance does not improve after 5 validation checks.

We also train a set of single-task models, which are configured and trained identically, but share no parameters. To allow for fair comparisons with the multi-task analogs, we do not tune parameter or training settings for each task, so these single-task models do not generally represent the state of the art for each task.
 
\paragraph{Sentence Representation Models}

Finally, we evaluate the following trained sentence-to-vector encoder models using our benchmark: average bag-of-words using GloVe embeddings (CBoW), Skip-Thought \citep{kiros2015skip}, InferSent \citep{DBLP:conf/emnlp/ConneauKSBB17}, DisSent \citep{nie2017dissent}, and GenSen \citep{subramanian2018large}. 
For these models, we only train task-specific classifiers on the representations they produce. 

\begin{table*}[t]
\centering \fontsize{8.4}{10.1}\selectfont \setlength{\tabcolsep}{0.5em}
\begin{tabular}{lrrrrrrrrrr}

\toprule

&& \multicolumn{2}{c}{\textbf{Single Sentence}} & \multicolumn{3}{c}{\textbf{Similarity and Paraphrase}} & \multicolumn{4}{c}{\textbf{Natural Language Inference}} \\
% \cline{3-4}
% \cline{5-7}
% \cline{8-11} \\ % TODO: The spacing doesn't work well here for some reason.
\textbf{Model} & \multicolumn{1}{c}{\textbf{Avg}} & \multicolumn{1}{c}{\textbf{CoLA}} & \multicolumn{1}{c}{\textbf{SST-2}} & \multicolumn{1}{c}{\textbf{MRPC}} & \multicolumn{1}{c}{\textbf{QQP}} & \multicolumn{1}{c}{\textbf{STS-B}} & \multicolumn{1}{c}{\textbf{MNLI}} & \multicolumn{1}{c}{\textbf{QNLI}} & \multicolumn{1}{c}{\textbf{RTE}} & \multicolumn{1}{c}{\textbf{WNLI}} \\
\midrule
\multicolumn{11}{c}{Single-Task Training} \\
\midrule

BiLSTM & 63.9 & 15.7 & 85.9 & 69.3/79.4 & 81.7/61.4 & 66.0/62.8 & 70.3/70.8 & 75.7 & 52.8 & \textbf{\underline{65.1}} \\

~~+ELMo & 66.4 & \textbf{\underline{35.0}} & \underline{90.2} & 69.0/80.8 & 85.7/65.6 & 64.0/60.2 & 72.9/73.4 & 71.7 & 50.1 & \textbf{\underline{65.1}} \\

~~+CoVe & 64.0 & 14.5 & 88.5 & \underline{73.4}/\underline{81.4} & 83.3/59.4 & \underline{67.2}/\underline{64.1} & 64.5/64.8 & 75.4 & \underline{53.5} & \textbf{\underline{65.1}} \\

~~+Attn & 63.9 & 15.7 & 85.9 & 68.5/80.3 & 83.5/62.9 & 59.3/55.8 & 74.2/73.8 & \underline{77.2} & 51.9 & \textbf{\underline{65.1}} \\

~~+Attn, ELMo & \underline{66.5} & \textbf{\underline{35.0}} & \underline{90.2} & 68.8/80.2 & \textbf{\underline{86.5}}/\textbf{\underline{66.1}} & 55.5/52.5 & \textbf{\underline{76.9}}/\textbf{\underline{76.7}} & 76.7 & 50.4 & \textbf{\underline{65.1}} \\

~~+Attn, CoVe & 63.2 & 14.5 & 88.5 & 68.6/79.7 & 84.1/60.1 & 57.2/53.6 & 71.6/71.5 & 74.5 & 52.7 & \textbf{\underline{65.1}} \\

\midrule
\multicolumn{11}{c}{Multi-Task Training} \\
\midrule

%BiLSTM & 63.5 & 24.0 & 85.8 & 71.9/82.1 & 80.2/59.1 & 68.8/67.0 & 65.8/66.0 & 71.1 & 46.8 & 63.7 \\
BiLSTM & 64.2 & 11.6 & 82.8 & 74.3/81.8 & 84.2/62.5 & 70.3/67.8 & 65.4/66.1 & 74.6 & 57.4 & \textbf{\underline{65.1}} \\

%~~+ELMo & 64.8 & \underline{27.5} & 89.6 & 76.2/83.5 & 78.5/57.8 & 67.0/65.9 & 67.1/68.0 & 66.7 & 55.7 & 62.3 \\
~~+ELMo & 67.7 & 32.1 & 89.3 & \textbf{\underline{78.0}}/\textbf{\underline{84.7}} & 82.6/61.1 & 67.2/67.9 & 70.3/67.8 & 75.5 & 57.4 & \textbf{\underline{65.1}} \\

%~~+CoVe & 62.2 & 16.2 & 84.3 & 71.8/80.0 & 82.0/59.1 & 68.0/67.1 & 65.3/65.9 & 70.4 & 44.2 & \textbf{\underline{65.1}} \\
~~+CoVe & 62.9 & 18.5 & 81.9 & 71.5/78.7 & \underline{84.9}/60.6 & 64.4/62.7 & 65.4/65.7 & 70.8 & 52.7 & \textbf{\underline{65.1}} \\

%~~+Attn & 65.7 & 0.0 & 85.0 & 75.1/\textbf{\underline{83.7}} & 84.3/63.6 & \underline{73.9}/\underline{71.8} & 72.2/72.1 & \underline{82.1} & \textbf{\underline{61.7}} & 63.7 \\
~~+Attn & 65.6 & 18.6 & 83.0 & 76.2/83.9 & 82.4/60.1 & 72.8/70.5 & 67.6/68.3 & 74.3 & 58.4 & \textbf{\underline{65.1}} \\

%~~+Attn, ELMo & \textbf{\underline{69.0}} & 18.9 & \textbf{\underline{91.6}} & \textbf{\underline{77.3}}/83.5 & \underline{85.3}/\underline{63.3} & 72.8/71.1 & \underline{75.6}/\underline{75.9} & 81.7 & 61.2 & \textbf{\underline{65.1}} \\
~~+Attn, ELMo & \textbf{\underline{70.0}} & \underline{33.6} & \textbf{\underline{90.4}} & \textbf{\underline{78.0}}/84.4 & 84.3/\underline{63.1} & \underline{74.2}/\underline{72.3} & \underline{74.1}/\underline{74.5} & \textbf{\underline{79.8}} & \underline{58.9} & \textbf{\underline{65.1}} \\

%~~+Attn, CoVe & 64.3 & 19.4 & 83.6 & 75.2/83.0 & 84.9/61.1 & 72.3/71.1 & 69.9/68.7 & 78.9 & 38.3 & \textbf{\underline{65.1}} \\
~~+Attn, CoVe & 63.1 & 8.3 & 80.7 & 71.8/80.0 & 83.4/60.5 & 69.8/68.4 & 68.1/68.6 & 72.9 & 56.0 & \textbf{\underline{65.1}} \\

\midrule
\multicolumn{11}{c}{Pre-Trained Sentence Representation Models} \\
\midrule

CBoW & 58.9 & 0.0 & 80.0 & 73.4/81.5 & 79.1/51.4 & 61.2/58.7 & 56.0/56.4 & 72.1 & 54.1 & \textbf{\underline{65.1}} \\

Skip-Thought & 61.3 & 0.0 & 81.8 & 71.7/80.8 & 82.2/56.4 & 71.8/69.7 & 62.9/62.8 & 72.9 & 53.1 & \textbf{\underline{65.1}} \\

InferSent & 63.9 & 4.5 & \underline{85.1} & 74.1/81.2 & 81.7/59.1 & 75.9/75.3 & 66.1/65.7 & 72.7 & 58.0 & \textbf{\underline{65.1}} \\

DisSent & 62.0 & 4.9 & 83.7 & 74.1/81.7 & 82.6/59.5 & 66.1/64.8 & 58.7/59.1 & 73.9 & 56.4 & \textbf{\underline{65.1}} \\

GenSen & \underline{66.2} & \underline{7.7} & 83.1 & \underline{76.6}/\underline{83.0} & \underline{82.9}/\underline{59.8} & \textbf{\underline{79.3}}/\textbf{\underline{79.2}} & \underline{71.4}/\underline{71.3} & \underline{78.6} & \textbf{\underline{59.2}} & \textbf{\underline{65.1}} \\

\bottomrule
\end{tabular}
\caption{Baseline performance on the GLUE task test sets. 
% Bold denotes the best result in each column; underline denotes best result per within a section. 
For MNLI, we report accuracy on the matched and mismatched test sets. For MRPC and Quora, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation. For CoLA, we report Matthews correlation. For all other tasks we report accuracy. All values are scaled by 100.
% \textit{Avg} is the macro-average of all metrics, weighting tasks equally and first averaging metrics within each task as needed. % SB: We say this already.
A similar table is presented on the online platform. % SB: The appendix shows this.
}
\label{tab:benchmark}
\end{table*}

\section{Benchmark Results}\label{sec:experiments}

We train three runs of each model and evaluate the run with the best macro-average development set performance (see Table \ref{tab:benchmark-dev} in Appendix \ref{sec:apdx_dev}). For single-task and sentence representation models, we evaluate the best run for each individual task. We present performance on the main benchmark tasks in \autoref{tab:benchmark}. 

% single task baselines
%In most cases, using multi-task training over single-task training yields better overall scores, particularly among the parameter-rich attention models.
We find that multi-task training yields better overall scores over single-task training amongst models using attention or ELMo.
Attention generally has negligible or negative aggregate effect in single task training, but helps in multi-task training.
We see a consistent improvement in using ELMo embeddings in place of GloVe or CoVe embeddings, particularly for single-sentence tasks.
Using CoVe has mixed effects over using only GloVe.

% pretrained models
Among the pre-trained sentence representation models, we observe fairly consistent gains moving from CBoW to Skip-Thought to Infersent and GenSen.
% DisSent ranks behind GenSen but fluctuates between Skip-Thought and InferSent in per-task ranking. % SB: Can cut if needed.
Relative to the models trained directly on the GLUE tasks, InferSent is competitive and GenSen outperforms all but the two best.

Looking at results per task, we find that the sentence representation models substantially underperform on CoLA compared to the models directly trained on the task. 
%Similarly, with the exception of InferSent, the sentence representation models are outperformed on SST by our BiLSTM and its non-CoVe variants. 
%These discrepancies indicate a need for better transfer methods for generalizing outside of the tasks a model was trained on and for task diversity in evaluation methods, as we have sought to do with GLUE.
On the other hand, for STS-B, models trained directly on the task lag significantly behind the performance of the best sentence representation model.
%, which we interpret as indicating the necessity of using transfer learning methods trained on data outside of the GLUE benchmark in order to solve it.
Finally, there are tasks for which no model does particularly well. On WNLI, no model exceeds most-frequent-class guessing (65.1\%) and we substitute the model predictions for the most-frequent baseline. On RTE and in aggregate, even our best baselines leave room for improvement.
These early results indicate that solving GLUE is beyond the capabilities of current models and methods.
%, and that training on auxiliary tasks seems a necessary and promising direction.

\begin{table*}[t]
\centering \fontsize{8.4}{10.1}\selectfont \setlength{\tabcolsep}{0.5em}
\begin{tabular}{lr@{\hskip 3em}rrrr@{\hskip 3em}rrrrrr}
 \toprule
 && \multicolumn{4}{l}{\textbf{~~Coarse-Grained}} & \multicolumn{6}{c}{\textbf{Fine-Grained}}\\ % SB: Something odd is going on with centering. Fixing manually for now.
\textbf{Model} & \textbf{All} & \textbf{LS} & \textbf{PAS} & \textbf{L} & \textbf{K} & \textbf{UQuant} & \textbf{MNeg} & \textbf{2Neg} & \textbf{Coref} & \textbf{Restr} & \textbf{Down} \\
\midrule
\multicolumn{12}{c}{Single-Task Training} \\
\midrule
BiLSTM & 21 & 25 & 24 & 16 & 16 & 70 & \underline{53} & 4 & 21 & -15 & \textbf{\underline{12}} \\
~~+ELMo & 20 & 20 & 21 & 14 & 17 & 70 & 20 & \textbf{\underline{42}} & 33 & -26 & -3 \\
~~+CoVe & 21 & 19 & 23 & 20 & \underline{18} & 71 & 47 & -1 & 33 & -15 & 8 \\
~~+Attn & 25 & 24 & 30 & 20 & 14 & 50 & 47 & 21 & \underline{38} & -8 & -3 \\
~~+Attn, ELMo & \textbf{\underline{28}} & \textbf{\underline{30}} & \textbf{\underline{35}} & \textbf{\underline{23}} & 14 & \textbf{\underline{85}} & 20 & \textbf{\underline{42}} & 33 & -26 & -3 \\
~~+Attn, CoVe & 24 & 29 & 29 & 18 & 12 & 77 & 50 & 1 & 18 & \underline{-1} & \textbf{\underline{12}} \\

\midrule
\multicolumn{12}{c}{Multi-Task Training} \\
\midrule
%BiLSTM & 19 & 16 & 22 & 16 & 17 & 71 & 35 & -8 & 26 & \textbf{\underline{0}} & 8 \\ 
%~~+ELMo & 19 & 15 & 21 & 17 & \textbf{\underline{21}} & 70 & \textbf{\underline{60}} & 15 & 26 & \textbf{\underline{0}} & \textbf{\underline{12}} \\
%~~+ELMo & 19 & 15 & 21 & 17 & \textbf{\underline{21}} & 70 & \textbf{\underline{60}} & 15 & 26 & \textbf{\underline{0}} & \textbf{\underline{12}} \\
%~~+CoVe & 17 & 15 & 21 & 14 & 16 & 50 & 31 & -8 & 25 & -15 & \textbf{\underline{12}} \\
%~~+Attn & \underline{25} & 23 & \underline{32} & \underline{19} & 16 & 58 & 26 & -5 & 28 & -1 & -20 \\
%~~+Attn, ELMo & 23 & \underline{24} & 30 & 17 & 13 & \underline{78} & 27 & \underline{37} & 30 & -15 & -20 \\
%~~+Attn, CoVe & 20 & 16 & 25 & 15 & 17 & \underline{78} & 37 & 14 & \underline{31} & -15 & 8 \\

BiLSTM & 20 & 13 & 24 & 14 & 22 & \underline{71} & 17 & -8 & 31 & -15 & 8 \\
~~+ELMo & 21 & \underline{20} & 21 & \underline{19} & 21 & \underline{71} & \textbf{\underline{60}} & 2 & 22 & 0 & \textbf{\underline{12}} \\
~~+CoVe & 18 & 15 & 11 & 18 & \textbf{\underline{27}} & 71 & 40 & \underline{7} & \textbf{\underline{40}} & 0 & 8 \\
~~+Attn & 18 & 13 & 24 & 11 & 16 & \underline{71} & 1 & -12 & 31 & -15 & 8 \\
~~+Attn, ELMo & \underline{22} & 18 & \underline{26} & 13 & 19 & 70 & 27 & 5 & 31 & -26 & -3 \\
~~+Attn, CoVe & 18 & 16 & 25 & 16 & 13 & \underline{71} & 26 & -8 & 33 & \textbf{\underline{9}} & 8 \\

\midrule
\multicolumn{12}{c}{Pre-Trained Sentence Representation Models} \\
\midrule
CBoW & 9 & 6 & 13 & 5 & 10 & 3 & 0 & \underline{13} & 28 & \underline{-15} & -11 \\
Skip-Thought & 12 & 2 & 23 & 11 & 9 & 61 & 6 & -2 & \underline{30} & \underline{-15} & 0 \\
InferSent & 18 & 20 & 20 & \underline{15} & 14 & 77 & 50 & -20 & 15 & \underline{-15} & -9 \\
DisSent & 16 & 16 & 19 & 13 & \underline{15} & 70 & 43 & -11 & 20 & -36 & -09 \\
GenSen & \underline{20} & \underline{28} & \underline{26} & 14 & 12 & \underline{78} & \underline{57} & 2 & 21 & \underline{-15} & \textbf{\underline{12}} \\
\bottomrule
\end{tabular}
\caption{Results on the diagnostic set. We report \(R_3\) coefficients between gold and predicted labels, scaled by 100.
The coarse-grained categories are \textit{Lexical Semantics} (\textbf{LS}),
\textit{Predicate-Argument Structure} (\textbf{PAS}),
\textit{Logic} (\textbf{L}),
and \textit{Knowledge and Common Sense} (\textbf{K}). Our example fine-grained categories are \textit{Universal Quantification} (\textbf{UQuant}),
\textit{Morphological Negation} (\textbf{MNeg}),
\textit{Double Negation} (\textbf{2Neg}),
\textit{Anaphora/Coreference} (\textbf{Coref}), \textit{Restrictivity} (\textbf{Restr}), and \textit{Downward Monotone} (\textbf{Down}).}
\label{tab:diagnostic}
\end{table*}

% \begin{figure}[t]
% \centering
% \begin{subfigure}{.9\linewidth}
%   \small
%   \centering
%   \begin{tabular}{ccccc}
%   \toprule
%   \textbf{Gold \textbackslash Prediction} & \textbf{All} & \textbf{E} & \textbf{C} & \textbf{N} \\
%   \midrule
%   \textbf{All} & & 65 & 16 & 19 \\
%   \textbf{E} & 42 & 34 & ~~3 & ~~4 \\
%   \textbf{C} & 23 & 11 & ~~8 & ~~4 \\
%   \textbf{N} & 35 & 19 & ~~5 & 11 \\
%  \bottomrule
% \end{tabular}
% %   \includegraphics[width=.9\linewidth]{figures/confusion}
%   \caption{Confusion matrix for BiLSTM +Attn (percentages).}
%   \label{fig:confusion}
% \end{subfigure}
% SB: We can re-introduce this in the CR.

% \centering
% \begin{subfigure}{.9\linewidth}
%   \small
%   \centering
%   \begin{tabular}{lccc}
%   \\
%   \toprule
%   \textbf{Model} & \textbf{E} & \textbf{C} & \textbf{N} \\
%   \midrule
%   BiLSTM & 71 & 16 & 13 \\
%   $\text{BiLSTM +Attn}^\mathcal{A}$ & \textbf{65} & 16 & \textbf{19} \\
%   BiLSTM +ELMo & 81 & ~~9 & 10 \\
%   BiLSTM +Cove & 75 & 13 & 13 \\
%   \midrule
%   CBoW & 84 & ~~7 & ~~9 \\
%   SkipThought & 80 & ~~8 & 12 \\
%   InferSent & 68 & \textbf{21} & 11 \\
%   DisSent & 73 & 18 & ~~8 \\
%   GenSen & 74 & 15 & 11 \\
%   \midrule
%   Gold & 42 & 23 & 35 \\
%   \bottomrule
%   \end{tabular}
% %   \includegraphics[width=.9\linewidth]{figures/labeldist}
%   \caption{Output class distributions (percentages). Bolded numbers are closest to the gold distribution.}
%   \label{fig:labeldist}
% \end{subfigure}
%   \caption{Partial output of GLUE's error analysis, aggregated across our models.}
%   \label{fig:diagnostic-charts}
% \end{figure}
% SB: We can re-introduce this in the CR.
% JM: sounds good. I'll wait to update the numbers until then.

\section{Analysis}

We analyze the baselines by evaluating each model's MNLI classifier on the diagnostic set to get a better sense of their linguistic capabilities.
Results are presented in \autoref{tab:diagnostic}.

\paragraph{Coarse Categories}
Overall performance is low for all models: The highest total score of 28 still denotes poor absolute performance.
Performance tends to be higher on Predicate-Argument Structure and lower on Logic, though numbers are not closely comparable across categories.
Unlike on the main benchmark, the multi-task models are almost always outperformed by their single-task counterparts.
This is perhaps unsurprising, since with our simple multi-task training regime, there is likely some destructive interference between MNLI and the other tasks.
The models trained on the GLUE tasks largely outperform the pretrained sentence representation models, with the exception of GenSen.
Using attention has a greater influence on diagnostic scores than using ELMo or CoVe, which we take to indicate that attention is especially important for generalization in NLI.

% JM: can add back in the CR when we add the tables back etc.
%\paragraph{Domain Shift \& Class Priors}
%GLUE's online platform also provides a submitted model's predicted class distributions and confusion matrices. 
%One point is immediately clear: All models severely under-predict \textit{neutral} and over-predict \textit{entailment}. This is perhaps indicative of the models' inability to generalize and adapt to new domains. We hypothesize that they learned to treat high lexical overlap as a strong sign of entailment, and that surgical addition of new information to the hypothesis (as in the case of neutral instances in the diagnostic set) might go unnoticed.
%Indeed, the attention-based model seems more sensitive to the neutral class, and is perhaps better at detecting small sets of unaligned tokens because it explicitly tries to model these alignments.
% Future training and data collection methods might want to account for this bias to guide their models to do deeper reasoning.

\paragraph{Fine-Grained Subcategories}

Most models handle universal quantification relatively well.
Looking at relevant examples, it seems that relying on lexical cues such as ``all'' often suffices for good performance.
Similarly, lexical cues often provide good signal in morphological negation examples.

We observe varying weaknesses between models.
Double negation is especially difficult for the GLUE-trained models that only use GloVe embeddings. 
This is ameliorated by ELMo, and to some degree CoVe.
%, perhaps because the translation and language modeling objectives teach models that phrases like ``not bad'' and ``okay'' have similar distributions.
Also, attention has mixed effects on overall results, and models with attention tend to struggle with downward monotonicity.
Examining their predictions, we found that the models are sensitive to hypernym/hyponym substitution and word deletion as a signal of entailment, but predict it in the wrong direction (as if the substituted/deleted word were in an upward monotone context).
This is consistent with recent findings by \citet{mccoy2019subsequence} that these systems use the subsequence relation between premise and hypothesis as a heuristic shortcut.
Restrictivity examples, which often depend on nuances of quantifier scope, are especially difficult for almost all models.

Overall, there is evidence that going beyond sentence-to-vector representations, e.g. with an attention mechanism, might aid performance on out-of-domain data, and that transfer methods like ELMo and CoVe encode linguistic information specific to their supervision signal.
However, increased representational capacity may lead to overfitting, such as the failure of attention models in downward monotone contexts.
We expect that our platform and diagnostic dataset will be useful for similar analyses in the future, so that model designers can better understand their models' generalization behavior and implicit knowledge. 

\section{Conclusion}\label{sec:conclusion}

We introduce GLUE, 
a platform and collection of resources for evaluating and analyzing natural language understanding systems. 
We find that, in aggregate, models trained jointly on our tasks see better performance than the combined performance of models trained for each task separately.
We confirm the utility of attention mechanisms and transfer learning methods such as ELMo in NLU systems, which combine to outperform the best sentence representation models on the GLUE benchmark, but still leave room for improvement.
When evaluating these models on our diagnostic dataset, we find that they fail (often spectacularly) on many linguistic phenomena, suggesting possible avenues for future work.
In sum, the question of how to design general-purpose NLU models remains unanswered, and we believe that GLUE can provide fertile soil for addressing this challenge.

\section*{Acknowledgments}
We thank Ellie Pavlick, Tal Linzen, Kyunghyun Cho, and Nikita Nangia for their comments on this work at its early stages, and we thank Ernie Davis, Alex Warstadt, and Quora's Nikhil Dandekar and Kornel Csernai for providing access to private evaluation data. 
This project has benefited from financial support to SB by Google, Tencent Holdings, and Samsung Research, and to AW from AdeptMind and an NSF Graduate Research Fellowship. 

\bibliography{iclr2019_conference}
\bibliographystyle{iclr2019_conference}

\appendix

\begin{table*}[t]
\centering \fontsize{8.4}{10.1}\selectfont \setlength{\tabcolsep}{0.5em}
\begin{tabular}{lrrrrrrrrrr}

\toprule

&& \multicolumn{2}{c}{\textbf{Single Sentence}} & \multicolumn{3}{c}{\textbf{Similarity and Paraphrase}} & \multicolumn{4}{c}{\textbf{Natural Language Inference}} \\
% \cline{3-4}
% \cline{5-7}
% \cline{8-11} \\ % TODO: The spacing doesn't work well here for some reason.
\textbf{Model} & \multicolumn{1}{c}{\textbf{Avg}} & \multicolumn{1}{c}{\textbf{CoLA}} & \multicolumn{1}{c}{\textbf{SST-2}} & \multicolumn{1}{c}{\textbf{MRPC}} & \multicolumn{1}{c}{\textbf{QQP}} & \multicolumn{1}{c}{\textbf{STS-B}} & \multicolumn{1}{c}{\textbf{MNLI}} & \multicolumn{1}{c}{\textbf{QNLI}} & \multicolumn{1}{c}{\textbf{RTE}} & \multicolumn{1}{c}{\textbf{WNLI}} \\
\midrule
\multicolumn{11}{c}{Single-Task Training} \\
\midrule

BiLSTM & 66.7 & 17.6 & 87.5 & 77.9/85.1 & 85.3/82.0 & 71.6/72.0 & 66.7 & 77.0 & 58.5 & 56.3 \\

~~+ELMo & 68.7 & 44.1 & 91.5 & 70.8/82.3 & 88.0/84.3 & 70.3/70.5 & 68.6 & 71.2 & 53.4 & 56.3 \\

~~+CoVe & 66.8 & 25.1 & 89.2 & 76.5/83.4 & 86.2/81.8 & 70.7/70.8 & 62.4 & 74.4 & 59.6 & 54.9 \\

~~+Attn & 66.9 & 17.6 & 87.5 & 72.8/82.9 & 87.7/83.9 & 66.6/66.7 & 70.0 & 77.2 & 58.5 & 60.6 \\

~~+Attn, ELMo & 67.9 & 44.1 & 91.5 & 71.1/82.1 & 87.8/83.6 & 57.9/56.1 & 72.4 & 75.2 & 52.7 & 56.3 \\

~~+Attn, CoVe & 65.6 & 25.1 & 89.2 & 72.8/82.4 & 86.1/81.3 & 59.4/58.0 & 67.9 & 72.5 & 58.1 & 57.7 \\

\midrule
\multicolumn{11}{c}{Multi-Task Training} \\
\midrule

BiLSTM  & 60.0 & 18.6 & 82.3 & 75.0/82.7 & 84.4/79.3 & 69.0/66.9 & 65.6 & 74.9 & 59.9 & 9.9 \\ 
~~+ELMo & 63.1 & 26.4 & 90.9 & 80.2/86.7 & 84.2/79.7 & 72.9/71.5 & 67.4 & 76.0 & 55.6 & 14.1 \\
~~+CoVe & 59.3 & 9.8 & 82.0 & 73.8/81.0 & 83.4/76.6 & 64.5/61.9 & 65.5 & 70.4 & 52.7 & 32.4 \\
~~+Attn & 60.5 & 15.2 & 83.1 & 77.5/85.1 & 82.6/77.2 & 72.4/70.5 & 68.0 & 73.7 & 61.7 & 9.9 \\
~~+Attn, ELMo & 67.3 &  36.7 & 91.1 & 80.6/86.6 & 84.6/79.6 & 74.4/72.9 & 74.6 & 80.4 & 61.4 & 22.5 \\
~~+Attn, CoVe & 61.4 & 17.4 & 82.1 & 71.3/80.1 & 83.4/77.7 & 68.6/66.7 & 68.2 & 73.2 & 58.5 & 29.6 \\

\midrule
\multicolumn{11}{c}{Pre-Trained Sentence Representation Models} \\
\midrule

 CBoW & 61.4 & 4.6 & 79.5 & 75.0/83.7 & 75.0/65.5 & 70.6/71.1 & 57.1 & 62.5 & 71.9 & 56.3 \\
Skip-Thought & 61.8 & 0.0 & 82.0 & 76.2/84.3 & 78.9/70.7 & 74.8/74.8 & 63.4 & 58.5 & 73.4 & 49.3 \\
InferSent & 65.7 & 8.6 & 83.9 & 76.5/84.1 & 81.7/75.9 & 80.2/80.4 & 67.8 & 63.5 & 71.5 & 56.3 \\
DisSent & 63.8 & 11.7 & 82.5 & 77.0/84.4 & 81.8/75.6 & 68.9/69.0 & 61.2 & 59.9 & 73.9 & 56.3 \\
GenSen & 67.8 & 10.3 & 87.2 & 80.4/86.2 & 82.6/76.6 & 81.3/81.8 & 71.4 & 62.5 & 78.4 & 56.3 \\

\bottomrule
\end{tabular}
\caption{Baseline performance on the GLUE tasks' development sets. 
For MNLI, we report accuracy averaged over the matched and mismatched test sets. For MRPC and QQP, we report accuracy and F1. For STS-B, we report Pearson and Spearman correlation. For CoLA, we report Matthews correlation. For all other tasks we report accuracy. All values are scaled by 100.
}
\label{tab:benchmark-dev}
\end{table*}

\section{Additional Benchmark Details}\label{sec:apdx_data}

%\subsection{Dataset Construction}

\paragraph{QNLI} 
To construct a balanced dataset, we select all pairs in which the most similar sentence to the question was \emph{not} the answer sentence, as well as an equal amount of cases in which the correct sentence was the most similar to the question, but another distracting sentence was a close second. Our similarity metric is based on CBoW representations with pre-trained GloVe embeddings. % TODO: L2 or Euclidean?
This approach to converting pre-existing datasets into NLI format is closely related to recent work by \citet{white2017inference}, as well as to the original motivation for textual entailment presented by \citet{dagan2006pascal}. Both argue that many NLP tasks can be productively reduced to textual entailment. 

% SB: This is in the main body.
% \paragraph{RTE} The classes for the included RTE datasets is either two way (\textit{entailment}, \textit{not\_entailment}) or three way (\textit{entailment}, \textit{neutral}, \textit{contradiction}). We convert all the data to a two-class split, where we collapse \textit{neutral} and \textit{contradiction} into \textit{not\_entailment}, for consistency.


\section{Additional Baseline Details}\label{sec:apdx_baselines}

\subsection{Attention Mechanism}

%For sentence-pair tasks, we also evaluate models that use bidirectional attention between the two sentences. By explicitly modeling the interaction between sentences, these models fall outside the sentence-to-vector paradigm. 
% We implement and train our models using AllenNLP \citep{Gardner2017AllenNLPAD}. 
% We follow standard practice to contextualize each token with attention. % Omer: cite Parikh et al in the EMNLP version. % Sam: Is this really related to Parikh's work? Closer to BiDAF and some later NLI models.
We implement our attention mechanism as follows:
given two sequences of hidden states $u_1, u_2, \dots, u_M$ and $v_1, v_2, \dots, v_N$, we first compute matrix $H$ where $H_{ij} = u_i \cdot v_j$.
For each $u_i$, we get attention weights $\alpha_{i}$ by taking a softmax over the $i^{th}$ row of $H$, and get the corresponding context vector $\tilde{v}_i = \sum_j \alpha_{ij} v_j $ by taking the attention-weighted sum of the $v_j$. 
We pass a second BiLSTM with max pooling over the sequence $[u_1; \tilde{v}_1], \dots [u_M ; \tilde{v}_M]$ to produce $u'$.
We process the $v_j$ vectors analogously to obtain $v'$.
Finally, we feed $[u'; v'; |u' - v'|; u' * v']$ into a classifier.

\subsection{Training}

We train our models with the BiLSTM sentence encoder and post-attention BiLSTMs shared across tasks, and classifiers trained separately for each task.
For each training update, we sample a task to train with a probability proportional to the number of training examples for each task.
We scale each task's loss inversely proportional to the number of examples for that task, which we found to improve overall performance.
We train our models with Adam \citep{kingma2014adam} with initial learning rate $10^{-3}$, batch size 128, and gradient clipping.
We use macro-average score over all tasks as our validation metric, and perform a validation check every 10k updates.  
We divide the learning rate by 5 whenever validation performance does not improve. 
We stop training when the learning rate drops below $10^{-5}$ or performance does not improve after 5 validation checks. 

\subsection{Sentence Representation Models}

We evaluate the following sentence representation models:
\begin{enumerate}
\item CBoW, the average of the GloVe embeddings of the tokens in the sentence.
\item Skip-Thought \citep{kiros2015skip}, a sequence-to-sequence(s) model trained to generate the previous and next sentences given the middle sentence. 
We use the original pre-trained model\footnote{\href{https://github.com/ryankiros/skip-thoughts}{\tt github.com/\allowbreak ryankiros/\allowbreak skip-thoughts}} trained on sequences of sentences from the Toronto Book Corpus (\citealt{zhu2015aligning}, TBC). 
\item InferSent \citep{DBLP:conf/emnlp/ConneauKSBB17}, a 
BiLSTM with max-pooling trained on MNLI and SNLI.
\item DisSent \citep{nie2017dissent}, a BiLSTM with max-pooling trained to predict the discourse marker (\textit{because}, \textit{so}, etc.) relating two sentences on data derived from TBC. 
We use the variant trained for eight-way classification.
\item GenSen \citep{subramanian2018large}, a sequence-to-sequence model trained on a variety of supervised and unsupervised objectives. 
We use the variant of the model trained on both MNLI and SNLI, the Skip-Thought objective on TBC, and a constituency parsing objective on the Billion Word Benchmark.
\end{enumerate}

We train task-specific classifiers on top of frozen sentence encoders, using the default parameters from SentEval. See \href{https://github.com/nyu-mll/SentEval}{https://github.com/nyu-mll/SentEval} for details and code.

\section{Development Set Results}\label{sec:apdx_dev}

The GLUE website limits users to two submissions per day in order to avoid overfitting to the private test data. To provide a reference for future work on GLUE, we present the best development set results achieved by our baselines in Table \ref{tab:benchmark-dev}.

\section{Benchmark Website Details}\label{sec:apdx_website}
\begin{figure*}[t]
\centering
\includegraphics[width=1\linewidth]{images/leaderboard.pdf}
\caption{The benchmark website leaderboard. An expanded view shows additional details about each submission, including a brief prose description and parameter count.}
\label{fig:leaderboard}
\end{figure*}
GLUE's online platform is built using React, Redux and TypeScript. We use Google Firebase for data storage and Google Cloud Functions to host and run our grading script when a submission is made. \autoref{fig:leaderboard} shows the visual presentation of our baselines on the leaderboard. 


\section{Additional Diagnostic Data Details}\label{sec:apdx_diagnostic}

The dataset is designed to allow for analyzing many levels of natural language understanding, from word meaning and sentence structure to high-level reasoning and application of world knowledge. To make this kind of analysis feasible, we first identify four broad categories of phenomena: Lexical Semantics, Predicate-Argument Structure, Logic, and Knowledge. However, since these categories are vague, we divide each into a larger set of fine-grained subcategories. Descriptions of all of the fine-grained categories are given in the remainder of this section. These categories are just one lens that can be used to understand linguistic phenomena and entailment, and there is certainly room to argue about how examples should be categorized, what the categories should be, etc. These categories are not based on any particular linguistic theory, but broadly based on issues that linguists have often identified and modeled in the study of syntax and semantics.
 
The dataset is provided not as a benchmark, but as an analysis tool to paint in broad strokes the kinds of phenomena a model may or may not capture, and to provide a set of examples that can serve for error analysis, qualitative model comparison, and development of adversarial examples that expose a model's weaknesses. Because the distribution of language is somewhat arbitrary, it will not be helpful to compare performance of the same model on different categories. Rather, we recommend comparing performance that different models score on the same category, or using the reported scores as a guide for error analysis.
 
We show coarse-grain category counts and label distributions of the diagnostic set in \autoref{tab:analysis-stats}.


\begin{table*}[t]
\centering \small
\begin{tabular}{lrrrr}
 \toprule
\textbf{Category} & \textbf{Count} & \textbf{\% Neutral} & \textbf{\% Contradiction} & \textbf{\% Entailment}  \\
\midrule
Lexical Semantics & 368 & 31.0 & 27.2 & 41.8 \\
Predicate-Argument Structure & 424 & 37.0 & 13.7 & 49.3 \\
Logic & 364 & 37.6 & 26.9 & 35.4 \\
Knowledge & 284 & 26.4 & 31.7 & 41.9 \\
\bottomrule
\end{tabular}
\caption{Diagnostic dataset statistics by coarse-grained category. Note that some examples may be tagged with phenomena belonging to multiple categories.
}
\label{tab:analysis-stats}
\end{table*}

\subsection{Lexical Semantics}
These phenomena center on aspects of word meaning.

\paragraph{Lexical Entailment} Entailment can be applied not only on the sentence level, but the word level. For example, we say ``dog'' lexically entails ``animal'' because anything that is a dog is also an animal, and ``dog'' lexically contradicts ``cat'' because it is impossible to be both at once. This relationship applies to many types of words (nouns, adjectives, verbs, many prepositions, etc.) and the relationship between lexical and sentential entailment has been deeply explored, e.g., in systems of natural logic. This connection often hinges on monotonicity in language, so many Lexical Entailment examples will also be tagged with one of the Monotone categories, though we do not do this in every case (see Monotonicity, under Logic).

\paragraph{Morphological Negation} This is a special case of lexical contradiction where one word is derived from the other: from ``affordable'' to ``unaffordable'', ``agree'' to ``disagree'', etc. We also include examples like ``ever'' and ``never''. We also label these examples with Negation or Double Negation, since they can be viewed as involving a word-level logical negation.

\paragraph{Factivity} Propositions appearing in a sentence may be in any entailment relation with the sentence as a whole, depending on the context in which they appear. In many cases, this is determined by lexical triggers (usually verbs or adverbs) in the sentence. For example,

\begin{itemize}
    \item ``I recognize that X'' entails ``X''.
    \item ``I did not recognize that X'' entails ``X''.
    \item ``I believe that X'' does not entail ``X''.
    \item ``I am refusing to do X'' contradicts ``I am doing X''.
    \item ``I am not refusing to do X'' does not contradict ``I am doing X''.
    \item ``I almost finished X'' contradicts ``I finished X''.
    \item ``I barely finished X'' entails ``I finished X''.
\end{itemize}

Constructions like ``I recognize that X'' are often called factive, since the entailment (of X above, regarded as a presupposition) persists even under negation. Constructions like ``I am refusing to do X'' above are often called implicative, and are sensitive to negation. There are also cases where a sentence (non-)entails the existence of an entity mentioned in it, for example ``I have found a unicorn'' entails ``A unicorn exists'' while ``I am looking for a unicorn'' doesn't necessarily entail ``A unicorn exists''. Readings where the entity does not necessarily exist are often called intensional readings, since they seem to deal with the properties denoted by a description (its intension) rather than being reducible to the set of entities that match the description (its extension, which in cases of non-existence will be empty).

We place all examples involving these phenomena under the label of Factivity. While it often depends on context to determine whether a nested proposition or existence of an entity is entailed by the overall statement, very often it relies heavily on lexical triggers, so we place the category under Lexical Semantics.

\paragraph{Symmetry/Collectivity} Some propositions denote symmetric relations, while others do not. For example, ``John married Gary'' entails ``Gary married John'' but ``John likes Gary'' does not entail ``Gary likes John''. Symmetric relations can often be rephrased by collecting both arguments into the subject: ``John met Gary'' entails ``John and Gary met''. Whether a relation is symmetric, or admits collecting its arguments into the subject, is often determined by its head word (e.g., ``like'', ``marry'' or ``meet''), so we classify it under Lexical Semantics.

\paragraph{Redundancy} If a word can be removed from a sentence without changing its meaning, that means the word's meaning was more-or-less adequately expressed by the sentence; so, identifying these cases reflects an understanding of both lexical and sentential semantics.

\paragraph{Named Entities} Words often name entities that exist in the world. There are many different kinds of understanding we might wish to understand about these names, including their compositional structure (for example, the ``Baltimore Police'' is the same as the ``Police of the City of Baltimore'') or their real-world referents and acronym expansions (for example, ``SNL'' is ``Saturday Night Live''). This category is closely related to World Knowledge, but focuses on the semantics of names as lexical items rather than background knowledge about their denoted entities.

\paragraph{Quantifiers} Logical quantification in natural language is often expressed through lexical triggers such as ``every'', ``most'', ``some'', and ``no''. While we reserve the categories in Quantification and Monotonicity for entailments involving operations on these quantifiers and their arguments, we choose to regard the interchangeability of quantifiers (e.g., in many cases ``most'' entails ``many'') as a question of lexical semantics.

\subsection{Predicate-Argument Structure}

An important component of understanding the meaning of a sentence is understanding how its parts are composed together into a whole. In this category, we address issues across that spectrum, from syntactic ambiguity to semantic roles and coreference.

\paragraph{Syntactic Ambiguity: Relative Clauses, Coordination Scope}

These two categories deal purely with resolving syntactic ambiguity. Relative clauses and coordination scope are both sources of a great amount of ambiguity in English.

\paragraph{Prepositional phrases} Prepositional phrase attachment is a particularly difficult problem that syntactic parsers in NLP systems continue to struggle with. We view it as a problem both of syntax and semantics, since prepositional phrases can express a wide variety of semantic roles and often semantically apply beyond their direct syntactic attachment.

\paragraph{Core Arguments} Verbs select for particular arguments, particularly subjects and objects, which might be interchangeable depending on the context or the surface form. One example is the ergative alternation: ``Jake broke the vase'' entails ``the vase broke'' but ``Jake broke the vase'' does not entail ``Jake broke''. Other rearrangements of core arguments, such as those seen in Symmetry/Collectivity, also fall under the Core Arguments label.

\paragraph{Alternations: Active/Passive, Genitives/Partitives, Nominalization, Datives} All four of these categories correspond to syntactic alternations that are known to follow specific patterns in English:
\begin{itemize}
    \item Active/Passive: ``I saw him'' is equivalent to ``He was seen by me'' and entails ``He was seen''.
    \item Genitives/Partitives: ``the elephant's foot'' is the same thing as ``the foot of the elephant''.
    \item Nominalization: ``I caused him to submit his resignation'' entails ``I caused the submission of his resignation''.
    \item Datives: ``I baked him a cake'' entails ``I baked a cake for him'' and ``I baked a cake'' but not ``I baked him''.
\end{itemize}

\paragraph{Ellipsis/Implicits} Often, the argument of a verb or other predicate is omitted (elided) in the text, with the reader filling in the gap. We can construct entailment examples by explicitly filling in the gap with the correct or incorrect referents. For example, the premise ``Putin is so entrenched within Russia’s ruling system that many of its members can imagine no other leader'' entails ``Putin is so entrenched within Russia’s ruling system that many of its members can imagine no other leader than Putin'' and contradicts ``Putin is so entrenched within Russia’s ruling system that many of its members can imagine no other leader than themselves.''

This is often regarded as a special case of anaphora, but we decided to split out these cases from explicit anaphora, which is often also regarded as a case of coreference (and attempted to some degree in modern coreference resolution systems).

\paragraph{Anaphora/Coreference} Coreference refers to when multiple expressions refer to the same entity or event. It is closely related to Anaphora, where the meaning of an expression depends on another (antecedent) expression in context. These two phenomena have significant overlap; for example, pronouns (``she'', ``we'', ``it'') are anaphors that are co-referent with their antecedents. However, they also may occur independently, such as coreference between two definite noun phrases (e.g., ``Theresa May ''and the ``British Prime Minister'') that refer to the same entity, or anaphora from a word like ``other'' which requires an antecedent to distinguish something from. In this category we only include cases where there is an explicit phrase (anaphoric or not) that is co-referent with an antecedent or other phrase. We construct examples for these in much the same way as for Ellipsis/Implicits.

\paragraph{Intersectivity} Many modifiers, especially adjectives, allow non-intersective uses, which affect their entailment behavior. For example:
\begin{itemize}
    \item Intersective: ``He is a violinist and an old surgeon'' entails ``He is an old violinist'' and ``He is a surgeon''.
    \item Non-intersective: ``He is a violinist and a skilled surgeon'' does not entail ``He is a skilled violinist''.
    \item Non-intersective: ``He is a fake surgeon'' does not entail ``He is a surgeon''.
\end{itemize}
Generally, an intersective use of a modifier, like ``old'' in ``old men'', is one which may be interpreted as referring to the set of entities with both properties (they are old and they are men). Linguists often formalize this using set intersection, hence the name.

Intersectivity is related to Factivity. For example, ``fake'' may be regarded as a counter-implicative modifier, and these examples will be labeled as such. However, we choose to categorize intersectivity under predicate-argument structure rather than lexical semantics, because generally the same word will admit both intersective and non-intersective uses, so it may be regarded as an ambiguity of argument structure.

\paragraph{Restrictivity} Restrictivity is most often used to refer to a property of uses of noun modifiers. In particular, a restrictive use of a modifier is one that serves to identify the entity or entities being described, whereas a non-restrictive use adds extra details to the identified entity. The distinction can often be highlighted by entailments:
\begin{itemize}
    \item Restrictive: ``I finished all of my homework due today'' does not entail ``I finished all of my homework''.
    \item Non-restrictive: ``I got rid of all those pesky bedbugs'' entails ``I got rid of all those bedbugs''.
\end{itemize}

Modifiers that are commonly used non-restrictively are appositives, relative clauses starting with ``which'' or ``who'', and expletives (e.g. ``pesky''). Non-restrictive uses can appear in many forms.

\subsection{Logic}

With an understanding of the structure of a sentence, there is often a baseline set of shallow conclusions that can be drawn using logical operators and often modeled using the mathematical tools of logic. Indeed, the development of mathematical logic was initially guided by questions about natural language meaning, from Aristotelian syllogisms to Fregean symbols. The notion of entailment is also borrowed from mathematical logic.

\paragraph{Propositional Structure: Negation, Double Negation, Conjunction, Disjunction, Conditionals}

All of the basic operations of propositional logic appear in natural language, and we tag them where they are relevant to our examples:
\begin{itemize}
    \item Negation: ``The cat sat on the mat'' contradicts ``The cat did not sit on the mat''.
    \item Double negation: ``The market is not impossible to navigate'' entails ``The market is possible to navigate''.
    \item Conjunction: ``Temperature and snow consistency must be just right'' entails ``Temperature must be just right''.
    \item Disjunction: ``Life is either a daring adventure or nothing at all'' does not entail, but is entailed by, ``Life is a daring adventure''.
    \item Conditionals: ``If both apply, they are essentially impossible'' does not entail ``They are essentially impossible''.
\end{itemize}

Conditionals are more complicated because their use in language does not always mirror their meaning in logic. For example, they may be used at a higher level than the at-issue assertion: ``If you think about it, it's the perfect reverse psychology tactic'' entails ``It's the perfect reverse psychology tactic''.

\paragraph{Quantification: Universal, Existential} Quantifiers are often triggered by words such as ``all'', ``some'', ``many'', and ``no''. There is a rich body of work modeling their meaning in mathematical logic with generalized quantifiers. In these two categories, we focus on straightforward inferences from the natural language analogs of universal and existential quantification:
\begin{itemize}
    \item Universal: ``All parakeets have two wings'' entails, but is not entailed by, ``My parakeet has two wings''.
    \item Existential: ``Some parakeets have two wings'' does not entail, but is entailed by, ``My parakeet has two wings''.
\end{itemize}

\paragraph{Monotonicity: Upward Monotone, Downward Monotone, Non-Monotone}

Monotonicity is a property of argument positions in certain logical systems. In general, it gives a way of deriving entailment relations between expressions that differ on only one subexpression. In language, it can explain how some entailments propagate through logical operators and quantifiers.

For example, ``pet'' entails ``pet squirrel'', which further entails ``happy pet squirrel''. We can demonstrate how the quantifiers ``a'', ``no'' and ``exactly one'' differ with respect to monotonicity:
\begin{itemize}
    \item ``I have a pet squirrel'' entails ``I have a pet'', but not ``I have a happy pet squirrel''.
    \item ``I have no pet squirrels'' does not entail ``I have no pets'', but does entail ``I have no happy pet squirrels''.
    \item ``I have exactly one pet squirrel'' entails neither ``I have exactly one pet'' nor ``I have exactly one happy pet squirrel''.
\end{itemize}

In all of these examples, ``pet squirrel'' appears in what we call the restrictor position of the quantifier. We say:
\begin{itemize}
    \item ``a'' is upward monotone in its restrictor: an entailment in the restrictor yields an entailment of the whole statement.
    \item ``no'' is downward monotone in its restrictor: an entailment in the restrictor yields an entailment of the whole statement in the opposite direction.
    \item ``exactly one'' is non-monotone in its restrictor: entailments in the restrictor do not yield entailments of the whole statement.
\end{itemize}
In this way, entailments between sentences that are built off of entailments of sub-phrases almost always rely on monotonicity judgments; see, for example, Lexical Entailment. However, because this is such a general class of sentence pairs, to keep the Logic category meaningful we do not always tag these examples with monotonicity.


\paragraph{Richer Logical Structure: Intervals/Numbers, Temporal}

Some higher-level facets of reasoning have been traditionally modeled using logic, such as actual mathematical reasoning (entailments based off of numbers) and temporal reasoning (which is often modeled as reasoning about a mathematical timeline).
\begin{itemize}
    \item Intervals/Numbers: ``I have had more than 2 drinks tonight'' entails ``I have had more than 1 drink tonight''.
    \item Temporal: ``Mary left before John entered'' entails ``John entered after Mary left''.
\end{itemize}

\subsection{Knowledge}

Strictly speaking, world knowledge and common sense are required on every level of language understanding for disambiguating word senses, syntactic structures, anaphora, and more. So our entire suite (and any test of entailment) does test these features to some degree. However, in these categories, we gather examples where the entailment rests not only on correct disambiguation of the sentences, but also application of extra knowledge, whether concrete knowledge about world affairs or more common-sense knowledge about word meanings or social or physical dynamics.

\paragraph{World Knowledge} In this category we focus on knowledge that can clearly be expressed as facts, as well as broader and less common geographical, legal, political, technical, or cultural knowledge. Examples:
\begin{itemize}
    \item ``This is the most oniony article I've seen on the entire internet'' entails ``This article reads like satire''.
    \item ``The reaction was strongly exothermic'' entails ``The reaction media got very hot''.
    \item ``There are amazing hikes around Mt. Fuji'' entails ``There are amazing hikes in Japan'' but not ``There are amazing hikes in Nepal''.
\end{itemize}

\paragraph{Common Sense} In this category we focus on knowledge that is more difficult to express as facts and that we expect to be possessed by most people independent of cultural or educational background. This includes a basic understanding of physical and social dynamics as well as lexical meaning (beyond simple lexical entailment or logical relations). Examples:
\begin{itemize}
    \item ``The announcement of Tillerson's departure sent shock waves across the globe'' contradicts ``People across the globe were prepared for Tillerson's departure''.
    \item ``Marc Sims has been seeing his barber once a week, for several years'' entails ``Marc Sims has been getting his hair cut once a week, for several years''.
    \item ``Hummingbirds are really attracted to bright orange and red (hence why the feeders are usually these colours)'' entails ``The feeders are usually coloured so as to attract hummingbirds''.
\end{itemize}


\end{document}
